Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[stateless_validation] Implement distribution of state witness parts #11115

Merged
merged 3 commits into from
Apr 24, 2024
Merged

[stateless_validation] Implement distribution of state witness parts #11115

merged 3 commits into from
Apr 24, 2024

Conversation

shreyan-gupta
Copy link
Contributor

This PR implements the following

  • Compressing state witness and creating Reed Solomon encoding parts out of it.
  • Sending the parts over the network to the appropriate part owners.
  • Forwarding parts received by owners to other chunk validators.

Future PRs would handle the following

  • Validation logic for the parts
  • Part management and reconstruction of state witness from parts.
  • Add tests after end to end implementation

@shreyan-gupta shreyan-gupta requested a review from a team as a code owner April 19, 2024 00:40
Shreyan Gupta added 2 commits April 18, 2024 21:20
@codecov Codecov
Copy link

codecov bot commented Apr 19, 2024

Codecov Report

Attention: Patch coverage is 18.85714% with 142 lines in your changes are missing coverage. Please review.

Project coverage is 71.02%. Comparing base (696190b) to head (2a64249).
Report is 1 commits behind head on master.

Files Patch % Lines
.../src/stateless_validation/state_witness_actions.rs 26.16% 78 Missing and 1 partial ⚠️
core/primitives/src/stateless_validation.rs 0.00% 51 Missing ⚠️
core/primitives/src/validator_signer.rs 0.00% 12 Missing ⚠️
Additional details and impacted files 
@@            Coverage Diff             @@
##           master   #11115      +/-   ##
==========================================
- Coverage   71.10%   71.02%   -0.09%     
==========================================
  Files         768      769       +1     
  Lines      153513   153814     +301     
  Branches   153513   153814     +301     
==========================================
+ Hits       109157   109242      +85     
- Misses      39918    40127     +209     
- Partials     4438     4445       +7
Flag Coverage Δ
backward-compatibility 0.24% <0.00%> (-0.01%) ⬇️
db-migration 0.24% <0.00%> (-0.01%) ⬇️
genesis-check 1.43% <0.00%> (-0.01%) ⬇️
integration-tests 36.86% <18.85%> (+0.02%) ⬆️
linux 69.41% <0.57%> (-0.12%) ⬇️
linux-nightly 70.48% <18.85%> (-0.12%) ⬇️
macos 54.18% <0.57%> (+1.55%) ⬆️
pytests 1.66% <0.00%> (-0.01%) ⬇️
sanity-checks 1.44% <0.00%> (-0.01%) ⬇️
unittests 66.67% <18.85%> (-0.12%) ⬇️
upgradability 0.29% <0.00%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

tayfunelmas
tayfunelmas reviewed Apr 19, 2024
View reviewed changes
core/primitives/src/stateless_validation.rs
@@ -102,10 +175,12 @@ pub struct ChunkStateWitness {
/// with chunk_header.prev_block_hash().
/// This is needed to validate signature when the previous block is not yet
/// available on the validator side (aka orphan state witness).
/// TODO(stateless_validation): Deprecate once we send state witness in parts.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you use a issue id #ID instead of stateless_validation unless it is already common to use this keyword for TODOs? for finding out remaining tasks in code for a specific issue.

Copy link
Contributor Author

@shreyan-gupta shreyan-gupta Apr 19, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes sure! Let me update and put the issue ID. Fwiw, this is just a short term thing till I get the full implementation in place. We used to put the same TODO(project) pattern for resharding

tayfunelmas
tayfunelmas reviewed Apr 19, 2024
View reviewed changes
core/primitives/src/stateless_validation.rs
#[derive(Debug, Clone, PartialEq, Eq, BorshSerialize, BorshDeserialize)]
pub struct PartialEncodedStateWitness {}
pub struct PartialEncodedStateWitness {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you add a brief comment about what this is, what this represents. Even it is clear now it may be helpful for future readers of the code.

Longarithm
Longarithm approved these changes Apr 24, 2024
View reviewed changes
Copy link
Member

@Longarithm Longarithm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving to unblock merge

@shreyan-gupta shreyan-gupta mentioned this pull request Apr 24, 2024
Closed
@shreyan-gupta shreyan-gupta added this pull request to the merge queue Apr 24, 2024
Merged via the queue into master with commit 6aa1aad Apr 24, 2024
25 of 29 checks passed
@shreyan-gupta shreyan-gupta deleted the shreyan/stateless_validation/send_parts branch April 24, 2024 21:40
@shreyan-gupta shreyan-gupta added the A-stateless-validation Area: stateless validation label Apr 26, 2024
jancionear pushed a commit to jancionear/nearcore that referenced this pull request May 13, 2024
@ecp88 @snyk-bot
[Snyk] Upgrade react-router-dom from 6.20.1 to 6.21.1 (near#10418)
c536f46 
<p>This PR was automatically created by Snyk using the credentials of a
real user.</p><br /><h3>Snyk has created this PR to upgrade
react-router-dom from 6.20.1 to 6.21.1.</h3>

:information_source: Keep your dependencies up-to-date. This makes it
easier to fix existing vulnerabilities and to more quickly identify and
fix newly disclosed vulnerabilities when they affect your project.
<hr/>

- The recommended version is **7 versions** ahead of your current
version.
- The recommended version was released **22 days ago**, on 2023-12-21.


<details>
<summary><b>Release notes</b></summary>
<br/>
  <details>
    <summary>Package name: <b>react-router-dom</b></summary>
    <ul>
      <li>
        <b>6.21.1</b> - 2023-12-21
      </li>
      <li>
        <b>6.21.1-pre.0</b> - 2023-12-21
      </li>
      <li>
        <b>6.21.0</b> - 2023-12-13
      </li>
      <li>
        <b>6.21.0-pre.3</b> - 2023-12-06
      </li>
      <li>
        <b>6.21.0-pre.2</b> - 2023-12-05
      </li>
      <li>
        <b>6.21.0-pre.1</b> - 2023-12-05
      </li>
      <li>
        <b>6.21.0-pre.0</b> - 2023-12-05
      </li>
      <li>
        <b>6.20.1</b> - 2023-12-01
      </li>
    </ul>
from <a
href="https://snyk.io/redirect/github/remix-run/react-router/releases">react-router-dom
GitHub release notes</a>
  </details>
</details>


<details>
  <summary><b>Commit messages</b></summary>
  </br>
  <details>
    <summary>Package name: <b>react-router-dom</b></summary>
    <ul>
<li><a
href="https://snyk.io/redirect/github/remix-run/react-router/commit/08cda17f450ebe19481be3fc080d243ec5ef509f">08cda17</a>
chore: Update version for release (near#11132)</li>
<li><a
href="https://snyk.io/redirect/github/remix-run/react-router/commit/14df5db772c1ec92fd680b7646c8865d5caf1a84">14df5db</a>
Exit prerelease mode</li>
<li><a
href="https://snyk.io/redirect/github/remix-run/react-router/commit/c55f8417e17491e8e4f5626680bc4c9b8d0f9ed9">c55f841</a>
Update release notes</li>
<li><a
href="https://snyk.io/redirect/github/remix-run/react-router/commit/e5d73cf2251bf99d3113695a57c34c84e0ac92e5">e5d73cf</a>
chore: Update version for release (pre) (near#11131)</li>
<li><a
href="https://snyk.io/redirect/github/remix-run/react-router/commit/a7ec1c50756172b92aeff2225cbedca7254c63b7">a7ec1c5</a>
Enter prerelease mode</li>
<li><a
href="https://snyk.io/redirect/github/remix-run/react-router/commit/ba99ec567e1899f811973d99ca804dc620a9cfae">ba99ec5</a>
Merge branch &#x27;main&#x27; into release-next</li>
<li><a
href="https://snyk.io/redirect/github/remix-run/react-router/commit/8bb3ffdf3225dafd2c8df11a27141b0745fcc647">8bb3ffd</a>
Fix typo in changeset</li>
<li><a
href="https://snyk.io/redirect/github/remix-run/react-router/commit/0f04d11d152a2d7d7ebace786bc7c2fd30d55fbc">0f04d11</a>
Fix issues with partial hydration combined with route.lazy (near#11121)</li>
<li><a
href="https://snyk.io/redirect/github/remix-run/react-router/commit/cc4436c544300a09d9d95c5f733d3b00fd7d426f">cc4436c</a>
Update changelogs for useResolvedSplat example</li>
<li><a
href="https://snyk.io/redirect/github/remix-run/react-router/commit/d9b36f6e16a7b437be30faf7fdd8154cff40e9cc">d9b36f6</a>
Update useResolvedPath docs</li>
<li><a
href="https://snyk.io/redirect/github/remix-run/react-router/commit/f8c54a89f604185e8462a7768d0172a670beae6d">f8c54a8</a>
chore: format</li>
<li><a
href="https://snyk.io/redirect/github/remix-run/react-router/commit/abb23de1345349fc81a5beb6ff6697ea04816078">abb23de</a>
doc edited for unstable_useViewTransitionState (near#11117)</li>
<li><a
href="https://snyk.io/redirect/github/remix-run/react-router/commit/a4e91e089a730443f638742759e7dd335aea1399">a4e91e0</a>
chore: sort contributors list</li>
<li><a
href="https://snyk.io/redirect/github/remix-run/react-router/commit/2ea19faf9b47222a797037a63e61375622ad0fb6">2ea19fa</a>
docs: Add minor grammatical changes to &#x60;errorElement&#x60; docs for
better readability (near#11119)</li>
<li><a
href="https://snyk.io/redirect/github/remix-run/react-router/commit/87d5d6114420b3f00e156c04564c2c24496b6235">87d5d61</a>
Fix unit test (near#11115)</li>
<li><a
href="https://snyk.io/redirect/github/remix-run/react-router/commit/19e6398f20779dd34d8b4eca0f3605bf5831a7f0">19e6398</a>
Add v7_relativeSplatPath to createBrowserRouter docs</li>
<li><a
href="https://snyk.io/redirect/github/remix-run/react-router/commit/c292bdb9bd1171546b5937c6ff226a6f064c8652">c292bdb</a>
Merge branch &#x27;release-next&#x27; into dev</li>
<li><a
href="https://snyk.io/redirect/github/remix-run/react-router/commit/5567158e1f7ab96683bcb4c757e27b1c22ceb68f">5567158</a>
Merge branch &#x27;release-next&#x27;</li>
<li><a
href="https://snyk.io/redirect/github/remix-run/react-router/commit/69ba50e06633fd4add234fb47f2d49b0a5ee41f9">69ba50e</a>
chore: Update version for release (near#11114)</li>
<li><a
href="https://snyk.io/redirect/github/remix-run/react-router/commit/ffd4373737a5bda3ad0fcebfa4895fbd8038a504">ffd4373</a>
Exit prerelease mode</li>
<li><a
href="https://snyk.io/redirect/github/remix-run/react-router/commit/54ec39ee35cfde99ee7218e6aa2bd1b7d8afc787">54ec39e</a>
Allow persisted fetchers unmounted during submit to revalidate
(near#11102)</li>
<li><a
href="https://snyk.io/redirect/github/remix-run/react-router/commit/a76ee41e08f0b2a88557b48656ff2fc318f5c4d0">a76ee41</a>
Dedup relative path logic in resolveTo (near#11097)</li>
<li><a
href="https://snyk.io/redirect/github/remix-run/react-router/commit/ea0ffeef8a0c353f8ef402b5df2ac7c60a4d0b49">ea0ffee</a>
chore: Update version for release (pre) (near#11095)</li>
<li><a
href="https://snyk.io/redirect/github/remix-run/react-router/commit/fe3c071037b18f447dabb44fb24e4b1bce2a2a15">fe3c071</a>
Slight refactor to partial hydration to leverage state.initialized
properly (near#11094)</li>
    </ul>

<a
href="https://snyk.io/redirect/github/remix-run/react-router/compare/8b1ee67ebc00fc3e778d5e36fe9bca140b576b5c...08cda17f450ebe19481be3fc080d243ec5ef509f">Compare</a>
  </details>
</details>
<hr/>

**Note:** *You are seeing this because you or someone else with access
to this repository has authorized Snyk to open upgrade PRs.*

For more information: <img
src="https://api.segment.io/v1/pixel/track?data=eyJ3cml0ZUtleSI6InJyWmxZcEdHY2RyTHZsb0lYd0dUcVg4WkFRTnNCOUEwIiwiYW5vbnltb3VzSWQiOiIyODZlOTdmZi04MTUxLTRiMWYtODM2Zi0zNzJiMTg4NzZjZWIiLCJldmVudCI6IlBSIHZpZXdlZCIsInByb3BlcnRpZXMiOnsicHJJZCI6IjI4NmU5N2ZmLTgxNTEtNGIxZi04MzZmLTM3MmIxODg3NmNlYiJ9fQ=="
width="0" height="0"/>

🧐 [View latest project
report](https://app.snyk.io/org/ecp88/project/98480bdc-d80b-4fd1-89d7-c4c56a706763?utm_source&#x3D;github&amp;utm_medium&#x3D;referral&amp;page&#x3D;upgrade-pr)

🛠 [Adjust upgrade PR
settings](https://app.snyk.io/org/ecp88/project/98480bdc-d80b-4fd1-89d7-c4c56a706763/settings/integration?utm_source&#x3D;github&amp;utm_medium&#x3D;referral&amp;page&#x3D;upgrade-pr)

🔕 [Ignore this dependency or unsubscribe from future upgrade
PRs](https://app.snyk.io/org/ecp88/project/98480bdc-d80b-4fd1-89d7-c4c56a706763/settings/integration?pkg&#x3D;react-router-dom&amp;utm_source&#x3D;github&amp;utm_medium&#x3D;referral&amp;page&#x3D;upgrade-pr#auto-dep-upgrades)

<!---
(snyk:metadata:{"prId":"286e97ff-8151-4b1f-836f-372b18876ceb","prPublicId":"286e97ff-8151-4b1f-836f-372b18876ceb","dependencies":[{"name":"react-router-dom","from":"6.20.1","to":"6.21.1"}],"packageManager":"npm","type":"auto","projectUrl":"https://app.snyk.io/org/ecp88/project/98480bdc-d80b-4fd1-89d7-c4c56a706763?utm_source=github&utm_medium=referral&page=upgrade-pr","projectPublicId":"98480bdc-d80b-4fd1-89d7-c4c56a706763","env":"prod","prType":"upgrade","vulns":[],"issuesToFix":[],"upgrade":[],"upgradeInfo":{"versionsDiff":7,"publishedDate":"2023-12-21T17:00:03.440Z"},"templateVariants":[],"hasFixes":false,"isMajorUpgrade":false,"isBreakingChange":false,"priorityScoreList":[]})
--->

Co-authored-by: snyk-bot <snyk-bot@snyk.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@saketh-are saketh-are saketh-are left review comments

@tayfunelmas tayfunelmas tayfunelmas approved these changes

@Longarithm Longarithm Longarithm approved these changes

@pugachAG pugachAG Awaiting requested review from pugachAG

@robin-near robin-near Awaiting requested review from robin-near

Assignees
No one assigned
Labels
A-stateless-validation Area: stateless validation
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@shreyan-gupta @tayfunelmas @saketh-are @Longarithm